Tiga Arsitektur Transformer
Evolusi Model Bahasa Besar ditandai oleh Perubahan Paradigma: beralih dari model khusus tugas ke "Pra-pelatihan Terpadu" di mana satu arsitektur dapat menyesuaikan diri dengan berbagai kebutuhan NLP.
Di inti perubahan ini adalah mekanisme Self-Attention, yang memungkinkan model menimbang pentingnya kata-kata yang berbeda dalam suatu urutan:
$$Attention(Q, K, V) = softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
1. Encoder-Saja (BERT)
- Mekanisme:Modeling Bahasa yang Ditutupi (MLM).
- Perilaku: Konteks dua arah; model "melihat" seluruh kalimat sekaligus untuk memprediksi kata-kata tersembunyi.
- Terbaik Digunakan Untuk: Pemahaman Bahasa Alami (NLU), analisis sentimen, dan Pengenalan Entitas Terkenal (NER).
2. Decoder-Saja (GPT)
- Mekanisme:Pemodelan Otomatis Berbasis Rekursif.
- Perilaku: Pemrosesan kiri-ke-kanan; memprediksi token berikutnya hanya berdasarkan konteks sebelumnya (masking kausal).
- Terbaik Digunakan Untuk: Generasi Bahasa Alami (NLG) dan menulis kreatif. Ini merupakan dasar dari LLM modern seperti GPT-4 dan Llama 3.
3. Encoder-Decoder (T5)
- Mekanisme:Transformer Transfer Teks-ke-Teks.
- Perilaku: Encoder memproses string input menjadi representasi padat, dan decoder menghasilkan string tujuan.
- Terbaik Digunakan Untuk: Terjemahan, ringkasan, dan tugas kesamaan.
Wawasan Utama: Dominasi Decoder
Industri telah banyak beralih ke arsitektur Decoder-saja karena hukum skalanya yang lebih unggul dan kemampuan pemikiran muncul dalam skenario zero-shot.
Dampak Jendela Konteks VRAM
Pada model Decoder-saja, KV Cache tumbuh secara linier dengan panjang urutan. Jendela konteks 100k membutuhkan VRAM jauh lebih besar dibandingkan jendela 8k, membuat penggunaan lokal model konteks panjang menjadi sulit tanpa kuantisasi.
TERMINALbash — 80x24
> Ready. Click "Run" to execute.
>
Question 1
Why did the industry move from BERT-style encoders to GPT-style decoders for Large Language Models?
Question 2
Which architecture treats every NLP task as a "text-to-text" problem?
Challenge: Architectural Bottlenecks
Analyze deployment constraints based on architecture.
If you are building a model for real-time document summarization where the input is very long, explain why a Decoder-only model might be preferred over an Encoder-Decoder model in modern deployments.
Step 1
Identify the architectural bottleneck regarding context processing.
Solution:
Encoder-Decoders must process the entire long input through the encoder, then perform cross-attention in the decoder, which can be computationally heavy and complex to optimize for extremely long sequences. Decoder-only models process everything uniformly. With modern techniques like FlashAttention and KV Cache optimization, scaling the context window in a Decoder-only model is more streamlined and efficient for real-time generation.
Encoder-Decoders must process the entire long input through the encoder, then perform cross-attention in the decoder, which can be computationally heavy and complex to optimize for extremely long sequences. Decoder-only models process everything uniformly. With modern techniques like FlashAttention and KV Cache optimization, scaling the context window in a Decoder-only model is more streamlined and efficient for real-time generation.
Step 2
Justify the preference using Scaling Laws.
Solution:
Decoder-only models have demonstrated highly predictable performance improvements (Scaling Laws) when increasing parameters and training data. This massive scale unlocks "emergent abilities," allowing a single Decoder-only model to perform zero-shot summarization highly effectively without needing the task-specific fine-tuning often required by smaller Encoder-Decoder setups.
Decoder-only models have demonstrated highly predictable performance improvements (Scaling Laws) when increasing parameters and training data. This massive scale unlocks "emergent abilities," allowing a single Decoder-only model to perform zero-shot summarization highly effectively without needing the task-specific fine-tuning often required by smaller Encoder-Decoder setups.